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Laura  H.  Baldwin1 
The  RAND  Corporation 

Air  Force  Service  Procurement 
Approaches  for  Measurement  and  Management2 

Before  the  Committee  on  Armed  Services 
Panel  on  Defense  Acquisition  Reform 
United  States  House  of  Representatives 

April  23,  2009 

Chairman  and  distinguished  members:  Thank  you  for  inviting  me  to  testify  at  this  panel  on 
“Measuring  Value  and  Risk  in  Service  Contracts.”  I  am  a  senior  economist  at  the  RAND 
Corporation  and  currently  serve  as  the  Director  of  the  Resource  Management  Program  within 
Project  AIR  FORCE. 

For  more  than  a  decade,  RAND  has  conducted  a  broad  portfolio  of  research  on  implementing 
best  purchasing  and  supply  management  practices  within  the  Department  of  Defense.  My 
testimony  today  will  describe  research  I  co-led  (along  with  my  colleague  John  Ausink  who  is  here 
today)  that  examined  successful  commercial  practices  for  managing  services  acquisition  and  that 
focused  on  performance  metrics  to  improve  contract  outcomes.  That  research  was  sponsored  by 
the  Air  Force  Deputy  Assistant  Secretary  for  Contracting  (SAF/AQC)  and  conducted  during  fiscal 
year  2003. 

Here  I  begin  by  providing  some  context  for  the  research,  followed  by  a  discussion  of  what  we  did, 
what  we  found,  and  how  it  is  relevant  to  the  U.S.  Air  Force. 

In  terms  of  context,  the  Department  of  Defense  purchases  many  services,  including  such 
commercial  services  as  building  maintenance,  grounds-keeping,  and  janitorial  services  for  its 
facilities  and  installations;  such  professional  services  as  consulting  and  engineering  support;  and 
such  weapon  system-oriented  services  as  research  and  development,  test  and  evaluation,  and 
maintenance  and  modification  activities.  Such  services  totaled  nearly  $100  billion  of  DoD’s 
budget  at  the  time  of  this  research. 


1  The  opinions  and  conclusions  expressed  in  this  testimony  are  the  author's  alone  and  should  not  be 
interpreted  as  representing  those  of  RAND  or  any  of  the  sponsors  of  its  research.  This  product  is  part  of  the 
RAND  Corporation  testimony  series.  RAND  testimonies  record  testimony  presented  by  RAND  associates  to 
federal,  state,  or  local  legislative  committees;  government-appointed  commissions  and  panels;  and  private 
review  and  oversight  bodies.  The  RAND  Corporation  is  a  nonprofit  research  organization  providing  objective 
analysis  and  effective  solutions  that  address  the  challenges  facing  the  public  and  private  sectors  around  the 
world.  RAND’s  publications  do  not  necessarily  reflect  the  opinions  of  its  research  clients  and  sponsors. 

2  This  testimony  is  available  for  free  download  at  http://www.rand.org/pubs/testimonies/CT328/. 
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Through  the  FY02-04  National  Defense  Authorization  Acts,  Congress  required  the  DoD  to  take 
steps  to  change  how  it  purchases  services  in  order  to  decrease  service  costs,  including  broadly 
implementing  performance-based  services  contracts  based  on  measurable  performance 
standards  and  a  new  management  structure  for  services  acquisitions. 

As  a  result,  the  Air  Force  established  a  Program  Executive  Officer  (PEO)  for  Combat  and  Mission 
Support  (AFPEO/CM)  who  is  responsible  for  management  and  oversight  of  a  well-defined 
portfolio  of  Air  Force  services  acquisition  activities.  This  office  is  the  single  point  of  contact  for  Air 
Force  services  acquisition  inquiries  and  is  also  responsible  for  developing  long-range  plans  for 
cost-effective  acquisition  of  services,  including  implementing  performance-based  contracts. 

Turning  now  to  the  question  of  what  we  did,  to  monitor  compliance  with  statutory  requirements, 
respond  to  inquiries  about  specific  acquisitions,  and  effectively  manage  Air  Force  services 
acquisition  activities  and  organizations,  the  PEO  for  services  (AFPEO/CM)  needs  information. 
RAND  Project  AIR  FORCE  was  asked  to  develop  a  portfolio  of  “overarching”  measures  to  allow 
the  AFPEO/CM  to  assess  the  health  of  Air  Force  acquisition  activities,  diagnose  problems,  and 
target  improvement  efforts. 

To  develop  this  portfolio  of  metrics,  we  considered  the  experience  of  commercial  firms.  While  we 
attended  private-sector  conferences  on  services  acquisition  practices  and  reviewed  the  business 
literature  to  gather  data  for  our  research,  our  findings  are  primarily  based  on  a  series  of  interviews 
with  commercial-sector  purchasing  professionals  who  are  respected  by  their  peers  for 
successfully  creating  and  implementing  what  are  widely  accepted  as  best  purchasing  and  supply 
management  practices,  focusing  on  their  application  to  service  acquisitions.  These  were 
considered  to  be  emerging,  state  of  the  art  practices  in  2003. 

The  services  discussed  in  our  data  sources  are  similar  to  commercial-like  services  purchased  by 
the  Air  Force,  including  facilities  services  such  as  building  maintenance,  custodial  services,  and 
landscaping;  telecommunications  services  such  as  computing,  help  desks,  and  call  centers;  and 
other  support  services  such  as  human  resources,  consulting,  and  food  services. 

As  for  what  we  found,  our  primary  finding  is  that  commercial  firms  are  shifting  away  from 
uncoordinated  purchasing  at  lower  levels  of  the  organization  and  moving  toward  a  corporate 
approach  to  managing  their  service  acquisitions  in  much  the  same  way  as  they  acquire  direct 
materials.  Commercial  firms  are  increasingly  using  commodity  councils  (which  are  centralized 
cross-functional  teams)  to  develop  organization-wide  purchasing  and  supply  management 
strategies  for  services;  moreover,  such  firms  are  developing  and  using  performance  metrics, 
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similar  to  those  used  for  goods,  to  manage  their  purchased  services  and  their  purchasing 
organizations  that  support  those  strategies.  I  will  address  each  of  these  in  turn. 

When  developing  formal  strategies  for  individual  commodity  groups,  firms  consider  overarching 
corporate  objectives  (for  example,  increased  market  share  or  increased  revenue),  assessments 
of  internal  demands  for  services  (such  as  the  diversity  of  those  demands  and  whether  the  service 
is  provided  one  time  only  or  on  a  continuing  basis),  characteristics  of  the  purchased  services 
(such  as  the  consequence  of  poor  performance),  and  markets  for  those  services  (for  example, 
the  market  concentration).  Because  purchasing  and  supply  management  strategies  have  so 
many  facets,  commodity  councils  typically  include  representatives  of  user  groups,  experts  in  the 
particular  service  industry,  and  general  experts  in  purchasing.  Because  of  their  importance  to 
services  procurement,  industry  experts,  rather  than  procurement  personnel,  are  often  tapped  to 
lead  these  commodity  councils. 

Commercial  firms  find  it  difficult  to  choose  a  portfolio  of  metrics  that  can  inform  decisionmaking, 
select  appropriate  performance  thresholds,  and  populate  the  metrics  with  reliable  data.  Our 
interviewees  recommended  results-oriented  metrics — both  forward-looking  and  retrospective — 
that  focus  on  how  acquisition  activities  support  both  short-term  and  long  term-corporate 
objectives  to  manage  service  acquisition  activities.  The  most  frequent  results-oriented  metrics 
include  cost,  quality,  supplier  satisfaction,  implementation  of  new  initiatives  (such  as  supplier 
development),  and  special  interest  items  (such  as  small  business  participation).  Cost  metrics 
were  typically  oriented  toward  measuring  cost  savings  over  time,  often  adjusting  for  exogenous 
market  factors.  Quality,  which  is  more  difficult  to  quantify  for  services  than  for  goods,  was  typically 
measured  in  terms  of  user  satisfaction  or  service  reliability.  Supplier  satisfaction  was  used  as  a 
forward-looking  indicator  of  whether  the  buying  firm  will  be  able  to  continue  doing  business  with 
the  best  providers.  This  was  measured  primarily  through  surveys.  Metrics  for  implementing  new 
initiatives  were  a  mixture  of  forward-looking  process  measures  and  retrospective  outcome 
measures. 

In  addition  to  these  results-oriented  metrics,  commercial  firms  indicated  that  metrics  that  track 
internal  customer  satisfaction,  personnel  training  and  retention,  and  ethics  violations  were  also 
important  to  manage  the  internal  purchasing  organization.  Internal  customer  satisfaction  can  be 
measured  directly  through  surveys  or  indirectly  through  the  amount  of  purchasing  that  falls 
outside  corporate  strategies.  Training  and  retention  measure  building  and  retaining  the  high  level 
of  skills  required  to  successfully  manage  services  acquisition. 
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Developing  a  baseline  for  these  metrics  and  then  tracking  them  over  time  present  challenges  for 
many  firms.  Some  have  adopted  new  management  information  systems  to  collect  and  organize 
the  data  for  their  service  acquisitions  and  have  implemented  surveys  to  collect  additional  data 
such  as  supplier  satisfaction  and  customer  satisfaction  with  purchased  services,  the  purchasing 
organization,  and  its  processes.  Performance  in  selected  areas  such  as  cost,  quality,  and  supplier 
satisfaction  are  regularly  reported  up  to  top-level  executives. 

An  integrating  theme  is  that  commercial  firms  are  using  corporate  objectives  to  align  all  parts  of 
the  purchasing  process,  from  selection  of  the  purchasing  strategy,  to  selection  of  metrics, 
performance  thresholds,  purchasing  organizational  objectives,  and  personnel  incentives.  And  like 
the  corporate  objectives,  these  are  revisited  over  time  to  ensure  that  they  remain  relevant  and 
effective. 

As  for  how  these  findings  are  applicable  to  the  Air  Force,  although  the  Air  Force  is  not  a 
commercial  firm,  it  can  learn  from  commercial  firms’  experiences  in  managing  its  own  service 
acquisitions.  Our  findings  recommend  a  balanced  portfolio  of  performance  metrics  for  the 
services  PEO  (AFPEO/CM),  representing  both  short-  and  long-run  considerations  and  internal 
and  external  activities.  These  recommendations  are  based  on  the  six  major  categories  of  metrics 
discussed  earlier:  cost,  quality,  supplier  satisfaction,  new  initiatives,  special  interest,  and  internal 
management. 

As  with  commercial  firms,  the  Air  Force  will  find  it  challenging  to  populate  these  metrics.  Some  of 
the  required  data,  such  as  contract  costs,  are  in  Air  Force  contracting  data  systems.  But  other 
RAND  research  has  raised  concerns  about  how  useful  these  data  are  in  determining  what  types 
of  services  were  purchased  through  those  contracts.  At  the  time  of  this  research,  the  Air  Force 
did  not  have  many  of  the  required  data,  particularly  supplier  and  customer  satisfaction  data,  and 
would  need  to  implement  new  data  collection  procedures  to  effectively  implement  these  metrics. 
Given  the  cost  of  populating  metrics  with  accurate  data,  the  choice  of  metrics  should  be  revisited 
over  time  to  ensure  that  they  remain  useful  and  aligned  with  larger  objectives. 

More  broadly,  our  analyses  suggest  that  good  metrics  are  necessary  but  not  sufficient  to  extract 
maximum  value  from  purchased  services — that  they  are  most  powerful  when  implemented  in  a 
centralized  management  framework.  We  recommended  the  Air  Force  adopt  a  centralized, 
strategic  approach  linked  to  Air  Force  objectives  for  managing  its  purchased  services.  Given  the 
diversity  of  Air  Force  service  users  and  their  requirements,  we  recommended  the  Air  Force 
include  each  of  the  important  user  groups  in  the  process  of  developing  strategies  for  categories  of 
services,  as  well  as  industry  experts.  If  the  Air  Force  has  difficulty  hiring  such  industry  experts 
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from  the  private  sector,  the  industry  exchange  program  can  be  used  to  develop  this  kind  of 
expertise  over  time.  Finally,  the  Air  Force  will  need  to  reinforce  these  efforts  with  leadership 
support  and  incentives  that  are  aligned  with  Air  Force  objectives. 

In  summary,  our  research  indicates  that,  like  the  Air  Force,  commercial  firms  were  just  beginning 
to  pursue  strategic,  centralized  management  of  their  purchased  services,  and  they  based  their 
approaches  for  services  on  their  successful  approaches  for  goods.  Their  experiences  suggest 
that  a  multifaceted  management  approach — based  on  commodity  councils,  guided  by  a  balanced 
set  of  performance  metrics  that  reflect  important  dimensions  of  performance,  and  supported  by 
leadership  emphasis  and  incentives — can  lead  to  improved  service  outcomes  and  greater  value 
for  Air  Force  buying  organizations. 

This  concludes  my  remarks.  Thank  you,  and  I  welcome  any  questions  you  might  have. 
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